Goal-Guided Transformer-Enabled Reinforcement Learning for Efficient Autonomous Navigation

Despite some successful applications of goal-driven navigation, existing deep reinforcement learning (DRL)-based approaches notoriously suffers from poor data efficiency issue. One of the reasons is that the goal information is decoupled from the perception module and directly introduced as a condition of decision-making, resulting in the goal-irrelevant features of the scene representation playing an adversary role during the learning process. In light of this, we present a novel Goal-guided Transformer-enabled reinforcement learning (GTRL) approach by considering the physical goal states as an input of the scene encoder for guiding the scene representation to couple with the goal information and realizing efficient autonomous navigation. More specifically, we propose a novel variant of the Vision Transformer as the backbone of the perception system, namely Goal-guided Transformer (GoT), and pre-train it with expert priors to boost the data efficiency. Subsequently, a reinforcement learning algorithm is instantiated for the decision-making system, taking the goal-oriented scene representation from the GoT as the input and generating decision commands. As a result, our approach motivates the scene representation to concentrate mainly on goal-relevant features, which substantially enhances the data efficiency of the DRL learning process, leading to superior navigation performance. Both simulation and real-world experimental results manifest the superiority of our approach in terms of data efficiency, performance, robustness, and sim-to-real generalization, compared with other state-of-the-art (SOTA) baselines. The demonstration video (https://www.youtube.com/watch?v=aqJCHcsj4w0) and the source code (https://github.com/OscarHuangWind/DRL-Transformer-SimtoReal-Navigation) are also provided

A vehicle stability control strategy with adaptive neural network sliding mode theory based on system uncertainty approximation

Modelling uncertainty, parameter variation and unknown external disturbance are the major concerns in the development of an advanced controller for vehicle stability at the limits of handling. Sliding mode control (SMC) method has proved to be robust against parameter variation and unknown external disturbance with satisfactory tracking performance. But modelling uncertainty, such as errors caused in model simplification, is inevitable in model-based controller design, resulting in lowered control quality. The adaptive radial basis function network (ARBFN) can effectively improve the control performance against large system uncertainty by learning to approximate arbitrary nonlinear functions and ensure the global asymptotic stability of the closed-loop system. In this paper, a novel vehicle dynamics stability control strategy is proposed using the adaptive radial basis function network sliding mode control (ARBFN-SMC) to learn system uncertainty and eliminate its adverse effects. This strategy adopts a hierarchical control structure which consists of reference model layer, yaw moment control layer, braking torque allocation layer and executive layer. Co-simulation using MATLAB/Simulink and AMESim is conducted on a verified 15-DOF nonlinear vehicle system model with the integrated-electro-hydraulic brake system (I-EHB) actuator in a Sine With Dwell manoeuvre. The simulation results show that ARBFN-SMC scheme exhibits superior stability and tracking performance in different running conditions compared with SMC scheme

S-antigen specific T helper type 1 response is present in Behcet’s disease

PURPOSE: To investigate the frequency and phenotypic and functional characteristics of S-antigen (S-Ag) specific T cells in patients with Behcet's disease (BD). METHODS: Blood was taken from 23 active BD patients, 12 inactive BD patients, and 14 healthy controls. The clinical features of the patients were summarized. T cell response against 40 mixed S-Ag peptides was identified by interferon gamma (IFN-gamma) enzyme-linked immunospot assay (ELISPOT). CD69 and CD45RO were used to characterize the phenotype of S-Ag specific T cells. The functional property of S-Ag specific T cells was investigated by measuring the production of cytokines. RESULTS: Response to the mixed S-Ag peptides was found in 56.5% and 25% of active and inactive BD patients, respectively. The responsiveness to S-Ag peptides was unrelated to the clinical features of the patients. About 65.8% of IFN-gamma(+) CD4(+) T cells in active BD patients expressed CD69 and CD45RO concomitantly. S-Ag peptides significantly induced a production of IFN-gamma and tumor necrosis factor (TNF)-alpha but not interleukin (IL)-2, IL-4, and IL-17 by peripheral blood mononuclear cells (PBMCs) in active BD patients with a response to S-Ag. CONCLUSIONS: S-Ag specific T cells are present in certain active BD patients, and most of them are activated memory CD4(+) T cells. These T cells may be involved in the pathogenesis of BD via producing Th1-dominant cytokine

Toward personalized decision making for autonomous vehicles: a constrained multi-objective reinforcement learning technique

Reinforcement learning promises to provide a state-of-the-art solution to the decision making problem of autonomous driving. Nonetheless, numerous real-world decision making problems involve balancing multiple conflicting or competing objectives. In addition, passengers may typically prefer to explore diversified driving modes through their specific preferences (i.e., relative importance of different objectives). Taking into account these demands, traditional reinforcement learning algorithms with applications in personalized self-driving vehicles remain challenging. Consequently, here we present a novel constrained multi-objective reinforcement learning technique for personalized decision making in autonomous driving, with the goal of learning a single model for Pareto optimal policies across the space of all possible user preferences. Specifically, a nonlinear constraint incorporating a user-specified preference and a vectorized action–value function is introduced to ensure both diversity in learned decision behaviors and efficient alignment between the user-specified preference and the corresponding optimal policy. Additionally, a constrained multi-objective actor–critic approach is advanced to approximate the Pareto optimal policies for any user-specified preferences while adhering to the nonlinear constraint. Finally, the proposed personalized decision making scheme for autonomous driving is assessed in a highway on-ramp merging scenario with dynamic traffic flows. The results demonstrate the effectiveness of our method by comparing it with classical and state-of-the-art baselines.Agency for Science, Technology and Research (A*STAR)Nanyang Technological UniversityThis work was supported in part by A*STAR AME Young Individual Research Grant (No. A2084c0156), the MTC Individual Research Grants (No. M22K2c0079), the ANR-NRF joint grant (No.NRF2021-NRF-ANR003 HM Science), and SUG-NAP Grant of Nanyang Technological University, Singapore

Safe Decision-making for Lane-change of Autonomous Vehicles via Human Demonstration-aided Reinforcement Learning

Decision-making is critical for lane change in autonomous driving. Reinforcement learning (RL) algorithms aim to identify the values of behaviors in various situations and thus they become a promising pathway to address the decision-making problem. However, poor runtime safety hinders RL-based decision-making strategies from complex driving tasks in practice. To address this problem, human demonstrations are incorporated into the RL-based decision-making strategy in this paper. Decisions made by human subjects in a driving simulator are treated as safe demonstrations, which are stored into the replay buffer and then utilized to enhance the training process of RL. A complex lane change task in an off-ramp scenario is established to examine the performance of the developed strategy. Simulation results suggest that human demonstrations can effectively improve the safety of decisions of RL. And the proposed strategy surpasses other existing learning-based decision-making strategies with respect to multiple driving performances

Coliquefaction Reactivity of Biomass and Coal under Moderate Conditions. Part 2: Effect of Cornstalk Dosage on Viscosity of the Coliquefaction System

The viscous characteristics of the liquefaction system play a crucial rule during liquefaction, and these characteristics of the coal oil slurry (COS) have been studied. In this paper, the viscosity of the coal biomass oil slurry (CBOS) at three stages while not systematically studied by forerunners was investigated by the torque current in the autoclave during coliquefaction of coals and cornstalk. The results, therefore, show that, at the thermal calefaction stage, the torque current of the CBOS system, predicted with the exponential decay model, was larger than that of the COS. At the pyrogenation and hydrogenation stage, the torque current of COS varies slightly in the Shengli coal system compared to the Shendong coal system, which presents a maximum. At the cooling stage, the torque current of the COS system, increasing with the quantity of the CS, was greater than that of the CBOS system. The study, thus, in the thesis indicates three aspects for industrial practices as well as fundamental research: the pumpablility of feedstock in the calefaction stage, the anticarbonization in the pyrogenation and hydrogenation stage, and the kinetic foundation in coliquefaction

Hierarchical speed control for autonomous electric vehicle through deep reinforcement learning and robust control

Abstract For the speed control system of autonomous electric vehicle (AEV), challenge happens with how to determine an appropriate driving speed to satisfy the dynamic environment while resisting uncertainty and disturbance. Therefore, this paper proposes a robust optimal speed control approach based on hierarchical architecture for AEV through combining deep reinforcement learning (DRL) and robust control. In decision‐making layer, a deep maximum entropy proximal policy optimization (DMEPPO) algorithm is presented to obtain an optimal speed via dynamic environment information, heuristic target entropy and adaptive entropy constraint. In motion control layer, to track the learned optimal speed while resisting uncertainty and disturbance, a robust speed controller is designed by the linear matrix inequality (LMI). Finally, simulation experiment results show that the proposed robust optimal speed control scheme based on hierarchical architecture for AEV is feasible and effective